To address the length limit issue, the most straightforward method is to simply scale up the input context length.

In this work, we investigate whether analogical reasoning can enable in-context composition over composable elements of visual stimuli.

Then, to tackle the benchmark mentioned above, we present the Point-In-Context (PIC) to explore in-context learning for 3D point clouds.

Our theoretical analysis reveals that in this setting, in-context learning is more about identifying the task than about learning it, a result which is in line with a series of recent empirical findings.

Gradient techniques are inherently slow, sacrificing the "few-shot" quality

We study in-context learning for nonparametric regression with $α$-Hölder smooth regression functions, for some $α>0$. We prove that, with $n$ in-context examples and $d$-dimensional regression covariates, a pretrained transformer with $Θ(\log n)$ parameters and $Ω\bigl(n^{2α/(2α+d)}\log^3 n\bigr)$ pretraining sequences can achieve the minimax-optimal rate of convergence $O\bigl(n^{-2α/(2α+d)}\bigr)$ in mean squared error. Our result requires substantially fewer transformer parameters and pretraining sequences than previous results in the literature. This is achieved by showing that transformers are able to approximate local polynomial estimators efficiently by implementing a kernel-weighted polynomial basis and then running gradient descent.